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ABSTRACT 


The  dynamic  nature  of  hostile,  urban  environments  has  resulted  in  an  increased  interest  in 
novel  command  and  control  (C2)  technologies  and  associated  tactics,  techniques,  and 
procedures  (TTPs).  However,  the  introduction  of  new  technologies  to  support  C2 
significantly  impacts  performance  and  effectiveness  of  military  forces.  The  goal  of  our 
research  was  to  support  the  assessment  of  novel  Communications,  Command,  Control 
and  Intelligence  (C3I)  technologies  that  addressed  various  challenges  of  Military 
Operations  in  Urban  Terrain  (MOUT).  Our  novel  approach  combined  the  strengths  of 
field  assessment  with  virtual  and  constructive  simulations,  which  can  quantify  the  effects 
of  technologies  and  associated  TTPs  that  span  a  wide  range  of  capabilities;  including 
sensing,  situation  awareness/command  and  control  (SA/C2),  and  shaping  components. 
These  evaluation  capabilities  are  complemented  by  support  for  optimizing  TTPs  and 
organizational  structure  to  improve  performance  across  a  variety  of  metrics  (e.g.  mission 
completion  time,  execution  tempo,  team  task  load,  etc.).  Initial  assessments  were 
validated  by  comparing  results  from  field  studies  and  simulations,  which  confirmed  that 
our  approach  identified  force-multiplying  effects  of  emerging  technologies. 

Introduction 

Recent  events  in  Iraq  have  clearly  demonstrated  the  lethality  of  asymmetric  threats  on 
United  States  (US)  forces.  Furthermore,  MOUT  operations  (such  as  cordon  and  search  or 
patrols,  Table  1)  often  require  highly  decentralized  small  unit  operations  which  pose 
numerous  challenges  to  ground  forces  in  their  ability  to  successfully  complete  their 
mission.  Advances  in  C2  technologies  are  of  critical  importance  to  combat  the  increasing 
challenges  imposed  by  this  dynamic,  high-risk  environment.  However,  the  benefits  of 
advanced  technologies  are  not  necessarily  apparent  and  the  preeminent  use  of  these 
systems  is  not  well  known. 


Table  1  -  Examples  of  potential  MOUT  missions 


Fixed  Site  Security 

Dismounted  Patrol 

Cordon  and  Search 

Fortified  checkpoints 

Roving  patrols 

Recon  and  secure  routes 

Guard  towers 

Ad-hoc  checkpoints 

Area  security 

There  is  a  great  need  for  analysis  on  the  effectiveness  of  novel  C2  technologies  and  new 
TTPs  on  mission  performance.  Results  of  modeling  and  simulation  can  yield  significant 
evidence  on  the  most  effective  utilization  of  such  systems,  and  indications  of  where 
future  gains  are  to  be  made  in  the  development  of  these  technologies.  To  address  this 
need,  we  developed  an  approach  for  the  evaluation  of  technology  in  complex 
environments.  We  present  the  application  of  our  approach  on  the  Sensing  and  Patrolling 
Enhancements  Yielding  Effective  Security  (SPEYES)  system  and  discuss  the 
implications  of  our  analysis  on  the  development  of  future  technologies. 

In  the  following  sections,  we  present  relevant  background  on  technology  evaluation  and 
SPEYES.  We  then  describe  our  approach  to  provide  context  for  our  modeling  effort  and 
discuss  the  simulation  test  bed  we  used  for  our  experiments.  We  then  present  our 


computational  model,  discuss  the  design  of  our  experiments,  and  describe  the  results  and 
implications  of  our  analyses. 

Background 

Technology  Evaluation 

Figure  1  illustrates  how  new  technology  effects  can  manifest  themselves  at  multiple 
levels  in  a  complex  mission  (Linegang,  2006).  For  example,  a  new  information  system 
may  improve  the  performance  of  the  direct  users  of  that  system,  but  ultimately  produce 
information  bottlenecks  that  result  in  a  decrease  in  overall  “global  mission  performance” 
of  the  team;  or  conversely,  a  new  technology  may  yield  minimal  benefit  for  the  direct 
users  of  the  technology,  but  generate  substantial  benefit  for  indirect  users  in  another 
portion  of  the  team.  To  conduct  a  comprehensive  evaluation  of  new  technology  for 
mission  performance  in  a  command  and  control  environment,  one  must  be  able  measure 
performance  effects  at  several  levels  of  granularity,  including  1)  measurement  of  direct 
and  indirect  technology  impacts  on  command  and  control  decision-making,  and  2) 
measures  of  technology  impacts  on  team  performance  in  the  overall  mission. 


Figure  1.  Measurement  opportunities  for  evaluating  new  technology  effects  in  complex  mission 

environments. 


An  Example  Technology 

This  paper  describes  the  evaluation  of  SPEYES:  a  new  set  of  technologies  intended  to 
enhance  C3I  performance  in  infantry  soldier  operations.  SPEYES  technologies  include 
sensing  technologies  (Figure  2)  -  low  cost,  easily-emplaced,  camouflaged  sensors  (video, 
acoustic,  infrared,  motion);  technologies  to  support  situation  awareness/command  and 
control  -  tools  tailored  for  small  unit  operations  (planning,  blue  force  tracking,  resource 
management);  and  shaping  technologies,  such  as  tools  to  detect  vehicle-bom  improvised 
explosive  devices  (Popp,  2005). 


Figure  2  -  An  example  of  an  unattended  sensor 

The  SPEYES  technology  enablers  are  meant  to  increase  the  effectiveness  of  soldiers  with 
improved  threat  detection  and  prediction,  increased  situational  awareness,  and  improved 
decentralized  operations.  For  example,  the  speed  and  ability  of  US  forces  to  gather 
intelligence  by  the  combined  use  and  smart  placement  of  visual,  acoustic,  and  infrared 
sensors.  The  information  gathered  from  these  sensor  observations  are  used  by  troops  and 
commanders  to  gain  improved  situational  awareness  which  facilitates  resourceful 
patrolling.  Efficient  observations  combined  with  more  accurate  and  timely  orientation  is 
believed  to  lead  to  quicker,  more  effective  actions. 

The  Evaluation  Challenge 

SPEYES  technologies  are  expected  to  have  direct  impacts  on  command  and  control 
decision-making  for  infantry  operations  in  urban  environments,  ultimately  leading  to 
improved  mission  performance  in  these  environments.  Hence,  evaluating  SPEYES 
requires  the  ability  to  measure  command  and  control  effects  as  well  as  mission 
performance  effects  for  urban  infantry  operations  across  an  array  of  missions  with  a  wide 
range  of  variability.  There  are  three  general  test  environments  that  can  be  used  to 
evaluate  new  technologies:  1)  constructive  simulations,  2)  human- in-the-loop  (HITL) 
simulations,  and  3)  field  assessments  (Figure  3).  There  are  inherent  tradeoffs  between 
these  environments  for  an  evaluation  effort  such  as  SPEYES  (Table  2).  Because  of  these 
trade-offs,  no  one  environment  can  address  all  the  requirements  for  evaluation,  and 
technology  evaluation  efforts  increasingly  are  conducted  in  an  iterative  manner  across 
several  simulation  and  field  test  environments. 
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Figure  3  -  Measuring  technology  insertion  impacts 


Table  2.  Trade-offs  between  technology  evaluation  test  environments 


Field  Testing 

Virtual  Human-in-the-Loop 

Constructive  Simulation 

Strengths 

•  Valid  results 
(Evaluations  closely 
match  operational 
environment) 

•  Great  ability  to  observe 
complex  human- system 
performance  effects 

•  Some  control  over  variables 

•  Some  ability  to  observe 
human- system  interaction 

•  Easier  to  collect  measures 

•  Less  expensive  (after 
simulation  is  built) 

•  High  degree  of  control 
over  variables 

•  Easy  to  collect 
measures 

•  Inexpensive  (after 
simulation  is  built) 

•  Easy  to  evaluate  a  wide 
range  of  missions  and 
scenarios 

Weaknesses 

•  Limited  control  of 
variables 

•  Difficult  to  collect 

measures 

•  Expensive  &  limited  by 
human  SME  availability 
(difficult  to  evaluate  a 
wide  range  of  scenarios 
and  missions) 

•  Technology  and  training 
must  be  “field  ready” 

•  Need  to  validate  results  in 
field  environment 

•  Limited  by  human  SME 
availability  (difficult  to 
evaluate  a  wide  range  of 
scenarios  and  missions) 

•  Need  to  train  SME  operators 

•  Need  to  validate  results 
in  field  environment 

•  Cannot  observe 
complex  human-system 
interaction  effects. 

For  SPEYES  technology  evaluation,  the  high  expense,  lack  of  field-ready  technology, 
and  variability  of  expected  missions  and  scenarios  made  field  testing  an  inappropriate 
solution.  And  while  virtual  (HITL)  environments  offered  some  improvements,  it  too  had 
limited  ability  to  handle  the  variability  in  expected  missions  and  scenarios  for  SPEYES. 
Constructive  simulation  offered  the  most  viable  logistical  solution,  but  it  was  critical  for 
the  constructive  simulation  to  be  able  to  provide  measures  of  complex  human-system 


interaction  effects  for  command  and  control  and  mission  performance.  The  development 
of  an  agent-architecture  for  constructive  simulation  of  command  and  control  addressed 
this  requirement. 

Approach 

Modeling  and  simulation  can  support  experimental  design,  staffing,  technology  use, 
measurement,  and  can  focus  field  studies  on  the  most  relevant  parameters  of  a  system. 
Fortunately,  simulations  can  fulfill  a  key  complementary  role  in  the  testing  and 
evaluation  of  C2  technologies  and  associated  TTPs  before  they  are  inserted  in  the  field. 
By  utilizing  constructive  simulations,  we  are  able  to  test  a  multitude  of  alternative 
organizational  structures,  technologies,  and  decision-making  models.  Although  there  has 
been  some  work  regarding  the  use  of  agent-based  models  for  the  evaluation  of 
technology  (Bonabeau  et  al.,  2003;  Wijesekera,  D.,  2005),  less  focus  has  been  on  the 
impact  of  these  systems  on  performance  and  the  overall  effects  of  various  trade-offs  in 
technology  and  mission  parameters. 

Previously,  human-in-the-loop  and  agent  based  experiments  were  conducted  to  test 
various  concepts  including  organizational  congruence,  novel  network-centric  operations, 
strategy  selection  tradeoffs,  and  organizational  adaptation  (Kleinman  et  al.,  2003; 
Diedrich  et  al.,  2003;  Entin  et  al.,  2003;  Levchuk  et  al.,  2003).  Both  of  these  approaches  - 
human-in-the-loop  simulations  for  small-scale  organizations,  and  synthetic  agent 
simulations  -  have  limitations.  Large-scale  experiments  with  human  teams  are  expensive 
and  time  consuming.  Thus,  a  very  limited  number  are  conducted,  mainly  testing 
incremental  changes  to  the  current  doctrine  rather  than  revolutionary  ones.  Current 
synthetic  simulation  environments  capture  only  simplified  aspects  of  human  behavior,  so 
their  results  are  of  dubious  validity.  We  identified  a  need  for  low  cost,  highly  valid 
constructive  simulations  of  human  operators  and  teams.  These  produce  a  significant  gain 
in  research  impact  by  creating  executable  models  of  individual  and  team  decision  making 
that  can  replace  human  operators  in  organizations,  and  be  validated  against  them. 

To  demonstrate  and  evaluate  the  effects  of  technology  of  C2,  we  developed 
computational  models  of  distributed  decision  making,  command,  and  control  -  which 
were  used  to  simulate  the  processes  of  military  C3I  (e.g.,  decision-making  and 
information  processing)  during  mission  execution.  The  models  required  a  quantitative 
representation  of  military  missions  and  the  C2  organization.  These  missions  and 
organizational  structures  were  vetted  with  subject  matter  experts  (SME)  and 
representative  of  real  world  tasks.  Next,  we  identified  and  represented  the  interactions 
required  to  execute  those  missions.  Finally,  the  models  were  implemented  within 
constructive  and  virtual  simulation  environments  to  conduct  the  experiments. 

To  address  the  goals  of  our  research,  we  developed  a  flexible,  controllable,  distributed 
simulation/experimental  paradigm  suitable  for  examining  the  threats  and  mission 
essential  tasks  in  urban  environments  and  for  quantifying  the  potential  performance 
improvements  of  the  SPEYES  system  over  current  practices.  This  simulation  model 
captured  many  relevant  key  metrics  and  TTPs.  The  approach  utilized  the  Dynamic 


Distributed  Decision-making  (DDD)  Environment  as  both  a  virtual  and  agent-based 
constructive  simulation.  This  model  addresses  some  key  questions,  such  as: 

•  Mission  Scenarios  &  Human  Team/Organization 

o  What  tasks  need  to  be  accomplished? 
o  What  “players”  are  involved  in  the  mission? 

•  Technologies  Attributes 

o  Who  owns  the  technology  (Soldier,  Company,  Platoon  level)? 
o  What  can  it  do? 

■  What  functions  can  it  perform? 
o  What  operators  are  needed? 

•  Translate  into  simulation  variables 

o  What  are  the  capabilities  of  the  technology  we  can  model? 
o  What  are  the  effects  of  the  technology  for  the  model? 

•  Measures  of  performance  and  process 

o  Which  measures  are  applicable  for  both  the  field  and  simulation? 

As  field  experimentation  provides  information  on  what  to  measure,  such  as  information 
superiority,  sustainability,  mobility,  training,  and  survivability;  the  DDD  simulation 
provides  the  data  for  measurement.  This  results  in  interdependent  measures  such  as 
latency,  throughput,  and  resource  utilization  and  influences  dependencies  on  new 
technologies. 

Dynamic  Distributed  Decision-making  Environment  (DDD) 

The  DDD  (Kleinman  et  al.,  1996)  is  a  distributed  real-time  simulation  environment 
implementing  complex  synthetic  team  tasks  that  include  many  of  the  behaviors  at  the 
core  of  almost  any  team  mission:  assessing  the  situation,  planning  response  actions, 
gathering  information,  sharing  and  transferring  information,  allocating  resources  to 
accomplish  tasks,  coordinating  actions,  and  sharing  or  transferring  resources.  Successive 
DDD  generations  have  demonstrated  the  paradigm’s  flexibility  in  reflecting  different 
domains  and  scenarios  to  study  realistic  and  complex  team  decision-making.  The  DDD  is 
installed  and  used  at  more  than  twenty  sites  worldwide  by  researchers,  trainers,  and 
practitioners.  It  has  already  proven  an  effective  test  bed  for  conducting  experiments  in  a 
number  of  different  tactical  environments  including  the  Airborne  Warning  and  Control 
System,  Naval  Battle  Group,  Army  Ground  Maneuvers,  Army  Urban  Warfare/Special 
Operations,  Search  and  Rescue,  and  Joint  Peacekeeping  Operations.  Figure  4  shows  the 
modeling  process  and  examples  of  previous  applications  of  the  DDD. 


Air  Operations 


Figure  4  -  DDD  modeling  process  and  previous  applications  of  DDD 

In  general,  the  DDD  provides  an  extensive  set  of  capabilities  for  supporting  team 
experiments.  Although  many  multi-person  wargaming  simulators  exist  today,  the  DDD  is 
unique  in  its  flexibility.  Most  team  simulators  are  built  with  a  task,  information,  resource 
and  command  structure  that  replicates  a  specific  military  team  task.  The  DDD,  in 
contrast,  was  designed  to  capture  the  essential  elements  of  many  different  C2  team  tasks, 
and  allow  the  experimenter  to  vary  team  structure,  access  to  information,  and  control  of 
resources.  Because  it  is  rooted  in  a  strong  C2  performance  paradigm,  the  DDD  has  been 
able  to  provide  a  substantial  degree  of  control,  while  engaging  the  team  players  in  a  low- 
to-moderate  degree  of  realism,  in  very  different  environments.  The  reason  the  DDD 
simulator,  more  than  any  other  team  simulator,  has  been  able  to  step  “out  of  the  curve”  is 
because  it  infuses  its  team  processes  with  a  large  degree  of  functional  fidelity,  by 
focusing  on  the  team  functions  (e.g.,  communications,  coordination,  compensatory 
behaviors,  hierarchical  processes,  etc.),  as  opposed  to  the  physical  fidelity  that  comes 
very  often  at  the  expense  of  precise  control  of  variables.  The  gradual  introduction  of 
technology  into  the  simulator  (e.g.,  net  protocols,  multi-media  displays,  standard 
symbology,  verbal  and  electronic  communications,  realistic  force-on-force  scenarios, 
active  tactical  maps,  etc.)  further  increases  the  degree  of  realism  without  affecting  the 
DDD’s  ability  to  carefully  manipulate  a  large  set  of  external  and  internal  team  variables. 

The  DDD  has  successfully  supported  numerous  virtual  HITL  experiments,  but  as 
discussed  previously,  HITL  testing  was  not  a  viable  solution  for  the  SPEYES  problem. 
However,  the  results  from  HITL  testing  provided  a  model  for  an  agent-architecture  that 
could  enhance  the  DDD  environment  for  more  rigorous  constructive  simulation  to 
evaluate  technology  its  impact  on  C2. 


Learning  Complex  Human-System  Interaction  from  Virtual  Simulations 

In  recent  Adaptive  Architectures  for  Command  and  Control  (A2C2)  experiments 
(Diedrich  et  al.,  2003;  Entin  et  ah,  2003)  we  observed  that  human  operators  exercised 
several  trade-offs  between  accuracy  and  timeliness  of  task  execution  to  achieve  higher 
mission  effectiveness.  The  task  accuracy ,  or  effectiveness  of  task  execution,  is  achieved 
by  selecting  the  set  of  assets  to  execute  a  task  to  satisfy  all  of  its  resource  requirements. 
The  timeliness  refers  to  how  fast  a  task  can  be  executed.  However,  due  to  resource 
shortage,  not  all  of  the  most  efficient  resource  allocation  decisions  can  be  performed  in  a 
timely  manner  (Levchuk  et  al.,  2003).  To  execute  tasks  faster,  operators  may  use  fewer 
resources  than  required.  Although  this  improves  the  timeliness  of  a  tasks’  completion,  it 
reduces  the  execution  accuracy  (efficiency)  of  individual  tasks.  Another  component  of 
this  problem  is  the  cognitive  limitations  of  the  human  operators  who  monitor  and 
physically  control  resources.  The  timeliness  of  task  execution  depends  on  the  ability  of  an 
operator  to  perform  corresponding  control  functions  -  even  when  all  resources  are 
available  for  efficient  task  execution.  Due  to  the  constraints  on  cognitive  workload, 
operators  are  faced  with  limits  on  communication  as  well  as  the  number  of  resources  they 
can  control.  These  affordances  must  be  weighed  against  the  need  for  information 
sharing  and  inter-agent  communication  (required  synchronizing  actions  and  coordinating 
task  execution).  In  essence,  operators  must  account  for  how  fast  the  communication, 
control,  decision  making,  and  observation  activities  associated  with  the  resource 
allocation  decision  can  be  performed.  Thus,  human  operators  account  for  the 
efficiency/accuracy  of  the  individual  tasks’  execution,  the  time  when  these  tasks  will  be 
accomplished,  and  how  much  resource-task  allocation  decisions  impact  the  future 
operation  (activities/functions  execution)  of  decision-makers.  Since  the  task  timeliness  is 
affected  by  the  ability  of  decision-makers  to  perform  individual  functions/processes, 
operators  must  take  all  of  these  processes  into  account.  Previous  computational 
frameworks  modeled  team  coordination  and  individual  asset  control  workloads  (Levchuk 
et  al.,  2003).  However,  we  observed  that  human  operators  employed  different  trade-offs 
under  different  organizational  conditions  (Entin  et  al.,  2003;  Diedrich  et  al.,  2003; 
Kleinman  et  al.,  2003).  The  insights  from  these  HITL  experimental  results  provide  a 
model  for  representing  human  behaviors  in  the  form  of  simulation-based  agents. 

Agent-based  Modeling 

Our  computational  model  represents  human  operators  as  cooperative  agents  organized  in 
a  command  and  control  structure.  Individual  agents  make  decisions  that  affect  other  team 
members,  and  the  outcomes  of  those  decisions  are  influenced  by  the  position  of  agents  in 
the  organization  (command  structure)  and  access  to  resources  (control  structure).  Agents 
communicate  among  each  other  to  share  information,  submit  requests,  send  orders  to 
subordinates,  and  to  synchronize  the  assets  to  facilitate  task  execution.  Agents  perform 
four  main  functions  (also  called  processes ):  observation,  communication,  decision 
making,  and  resource  control  (Figure  5).  The  observation  function  is  related  to  gaining 
situation  understanding  via  observing  the  environment.  This  function  depends  on  the 
responsibility  of  the  corresponding  commander  (e.g.,  the  commander  needs  to  monitor 
the  situation  in  a  geographically  constrained  region,  or  is  responsible  for  a  function  such 


as  anti-air  warfare  and  therefore  is  monitoring  air  space).  The  observation  process  results 
in  improved  situation  awareness  for  an  agent  and  can  provide  information  about  what 
targets  appeared  or  disappeared  as  well  as  individual  and  shared  state.  This  can  include 
the  status  of  an  agent’s  assets  or  the  assets  of  other  agents,  the  status  of  plans,  mission 
tasks,  and  actions.  The  communication  function  allows  the  agent  to  communicate 
information,  requests,  and  orders  to  other  agents.  The  message  traffic  must  be  handled  by 
a  corresponding  communication  network.  Communication  is  also  needed  to  synchronize 
task  execution  by  multiple  assets  which  are  controlled  by  different  agents.  The  decision 
function  allows  the  agent  to  reason  about  resource  allocation  and  task  assignment 
problems.  These  decisions  are  based  on  the  situational  representation  that  the  agent  has 
built  from  observations  and  information  obtained  from  other  agents.  The  decisions  are 
constrained  by  the  role  of  the  agent  in  the  organization,  its  position  in  command 
hierarchy  (what  subordinates  the  agent  has),  and  its  controlled  resources/assets.  Finally, 
the  control  function  allows  the  agent  to  coordinate  its  assigned  assets  and  resources  to 
gather  information,  maneuver  in  the  environment,  execute  tasks  and  engage  the  enemy. 
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Figure  5  -  Four  main  functions/processes  of  computational  agent  model 

Analysis  and  Results 

While  innovative  technologies  and  new  operational  concepts  can  present  an  individual 
multiplier  effect,  the  goal  of  the  SPEYES  study  was  to  explore  and  determine  the  impact 
and  force  multiplier  payoffs  that  an  integrated  system  of  varied  technologies  might 
provide.  To  meet  this  objective,  we  aligned  operationally  relevant  measures  from  field 
experiments  to  present  a  realistic  assessment  of  the  utility  of  the  SPEYES  technologies 
and  to  analyze  new  TTPs  in  order  to  determine  the  overall  effectiveness  on  mission 
performance.  This  process  is  illustrated  in  Figure  6. 


We  conducted  quick  sensitivity  and  simulation  analyses  to  assess  the  impact  of  the 
SPEYES  system  on  the  efficiency  of  C2  processes  and  the  effectiveness  of  mission 
execution.  Operational  efficiency  measures  (measures  of  performance  or  MOP)  focused 
on  maintaining  the  same  degree  of  force  protection,  while  decreasing  the  number  of 
personnel,  amount  of  equipment,  and  cost/resources  needed  to  sustain  a  particular  level  of 
protection.  The  measures  of  effectiveness  (MOE)  were  defined  by  improved  force 
protection  and  incident  prevention.  In  addition  to  the  evaluation  of  technology  on  C2, 
our  integrated  framework  facilitates  the  assessment  of  the  impact  of  key  factors  on 
individual  workload,  team  processes,  and  mission  effects. 
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Figure  6  -  SPEYES  assessment  using  DDD 

Several  independent  and  dependent  metrics  from  field  experimentation  and  current 
military  doctrine  were  identified  for  the  evaluation  of  SPEYES  technologies,  which  we 
aligned  with  field  test  metrics  in  order  to  validate  the  model.  For  example,  the  overall 
task  processing  throughput,  which  is  the  ratio  of  the  number  of  tasks  processed  to  the 
mission  completion  time,  indicates  the  rate  at  which  activities  are  accomplished  by  the 
friendly  forces.  A  series  of  simulation  experiments  were  executed  for  different  packages 
of  technologies,  various  scenarios,  organizational  structures,  and  technology  utilization. 
These  were  defined  by  subject  matter  experts  where  technologies  were  thought  to  provide 
the  most  benefit.  The  scenarios  included  area  reconnaissance,  cordon  and  search, 
patrolling  routes,  and  establishing  observation  posts.  Various  tasks  of  interest  were 
identified  including: 

•  Snipers  and  insurgents 

•  Re-entering  a  cleared  building 

•  Reinforcement  of  cordon  area  from  outside 

•  Communications  to  unit  leader  from  inside  buildings  and  across  cordon  areas 


We  also  identified  the  top  threats  that  could  be  encountered  during  these  missions,  such 
as  improvised  explosive  devices  (IEDs),  rockets  and  mortars,  snipers,  vehicle  bom  IEDs 
(VBIEDs),  and  small  arms  fire. 

The  results  attained  were  promising  and  showed  that  the  SPEYES  system  achieved  force- 
multiplying  payoffs  using  an  array  of  integrated  SA/C2  technologies.  Specifically, 
technologies  provided  a  significant  reduction  in  mission  completion  time,  substantial 
reductions  in  friendly  casualties,  large  reductions  in  successful  enemy  sniper  and  RPG 
attacks,  significant  improvement  in  task  throughput,  and  troop  utilization  efficiencies. 
These  results  suggest  that  improvements  in  manning,  operational  performance,  and  C2 
can  be  achieved  by  integrating  individual  novel  technologies  into  the  SPEYES  system. 
As  part  of  the  analysis,  a  variety  of  challenges  were  generated  for  the  SPEYES  system, 
for  example: 

•  To  what  degree  will  surveillance  technology  improve  the  speed,  security,  and 
quality  of  searches? 

•  Will  utilizing  unattended  sensors  provide  sufficient  intelligence  to  allow  troops  to 
accomplish  other  tasks? 

We  hypothesized  that  using  certain  surveillance  technologies  would  increase  the  speed  of 
operation,  and  conducted  experiments  to  confirm  or  refute  these  assumptions.  Figure  7 
shows  an  example  of  an  initial  result  of  an  experiment  from  a  simple  version  of  the 
agents  we  developed,  illustrating  the  improvement  on  the  average  time  to  search  and 
secure  buildings  (immediate  and  second-order  effects)  with  and  without  the  SPEYES 
technologies  (six  building  company-sized  scenario). 


Search,  Engage  Enemy,  & 


■  W/O  SPEYES 
GW  SPEYES 

Figure  7  -  Technology  and  speed  effects 


Although  more  experiments  are  needed  for  a  complete  analysis,  the  initial  implications  of 
this  research  impacted  decisions  to  equip  units  in  the  Army  for  operations  in  Afghanistan 
and  the  Marines  for  operations  in  Iraq.  This  success  demonstrates  the  potential  utility  of 
our  approach  to  the  evaluation  of  technology  on  C2. 


Discussion  and  Conclusion 


Our  assessment  methodology  has  been  applied  to  evaluate  the  benefits  of  the  SPEYES 
system  -  a  ground-based,  decentralized,  multi-echelon  C3I  system  comprised  of 
emerging  and  existing  Sensing,  SA/C2,  and  Shaping  technologies,  and  tailored  for  squad 
level,  small  unit  forces.  Historically,  rapid  post-conflict  stability  has  been  attained 
through  high-troop  densities.  A  key  motivating  theme  leading  to  our  research  effort  was 
the  challenge  of  trying  to  enhance  the  effectiveness  with  a  limited  number  of  forces 
through  various  technologies.  Accordingly,  goals  of  the  SPEYES  experimentation 
included  quantifying  the  force  multiplying  effect  of  SPEYES  system,  assessing  its  effects 
on  mission  performance,  and  evaluating  the  impact  of  technologies  on  C2.  The 
performance  improvements  were  measured  in  terms  of  timeliness,  effectiveness,  and 
efficiency  of  operations. 

In  this  paper,  we  presented  a  computational  framework  and  approach  to  assess  the 
effectiveness  and  efficiency  of  C2  organizations  and  the  impacts  of  advanced 
technologies.  Our  process  draws  on  strengths  of  field  assessment,  virtual  simulations, 
and  agent-based  constructive  simulations.  Agents  are  based  on  a  quantitative 
representation  of  the  organization,  resources,  mission,  and  utilization  of  normative 
models  of  team  and  individual  decision-making  in  a  C2  environment.  One  of  the 
challenges  in  this  work  was  developing  representations  of  novel  systems  and 
technologies,  some  of  which  initially  existed  only  as  a  concept.  To  facilitate  the 
assessment,  our  models  have  been  implemented  within  the  DDD  simulation  environment. 
A  major  challenge  of  this  work  was  to  match  the  performance  of  synthetic  decision¬ 
making  (cognitive)  agent  models  with  real  human  C2  organization  performance.  We  are 
currently  implementing  a  more  flexible  agent  paradigm  that  models  technologies  at  a 
higher  physical  fidelity  (sensor  and  dynamic  models),  and  humans  at  a  higher  cognitive 
fidelity  (workload  and  process  based  models,  Levchuk  et  al.,  2002;  (Levchuk,  Yu, 
Pattipati,  and  Levchuk,  2003),  as  shown  in  Figure  8. 


Figure  8  -  Example  of  agent-based  constructive  model 

The  initial  agent-driven  simulation  results  indicated  that  integration  of  SPEYES  sensing, 
SA/C2,  and  shaping  technologies  provided  significant  performance  improvements  to  the 


force  across  all  measures.  Even  at  50%-reduced  force,  the  SPEYES  system  maintained 
significant  performance  improvements  over  regular  operations  with  a  full  force  and 
without  SPEYES,  thus  confirming  the  force  multiplier  effect  of  SPEYES  technologies. 
Those  findings  were  confirmed  by  human-in-the-loop  simulations. 

Throughout  this  paper,  we  have  described  the  evaluation  of  technology  as  it  applies  to 
mission  performance  and  C2.  More  generally,  the  evaluation  of  technology  is  becoming 
an  increasingly  important  area  for  purchasing,  funding,  and  engineering  decisions.  Table 
3  illustrates  how  the  results  of  SPEYES  could  be  used  to  aid  in  the  decision-making 
process.  In  addition,  this  framework  could  improve  the  ability  to  design  decision  support 
systems;  facilitate  important  acquisition  decisions  concerning  new  systems;  and  help 
develop  organizational  designs,  doctrine,  and  mission  plans  that  exploit  affordances  in 
socio-technical  systems  in  order  to  avoid  pitfalls. 


Table  3  -  Technology  evaluation  for  SPEYES 


Purchasing  and  funding  decisions 

Engineering  decisions 

In  terms  of  a  viable  solution, 
SPEYES  agent-based  simulations 
predict  that  the  technologies  can 
improve  local  mission  performance. 

In  terms  of  performance  objectives,  SPEYES 
simulations  provide  baseline  predictions  of 
magnitude  of  performance  improvements  expected 
from  these  technologies. 

In  terms  of  reliability,  SPEYES  simulations  assess 
different  tasks  and  missions,  providing  insight 
about  circumstances  that  may  challenge  systems. 
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Use  common  objectives 
Synchronize  assets 


TTP/CONOPS 

ROE 


asset  request 

- ► 


Action  execution 

synchronization 

•  Pursue 

L 

•  Attack 

1" 

T 


Coordination 


mission  update 
communicate  results 


(effect  on  task  completion  time)  •  (capability  for  other  tasks) 

asset  cost  - - 

(capability  per  selected  task) 
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Assessment  using  DDD: 
Measures  and  Metrics 


Technology 
Test  Plan 


I  nformation  Superiority 

Sustainability 

Mobility 

Training 

Survivability 

Etc. 


Simulation  Measures 


DDD 
Simulation 


I  nterdependent  measures 


Log  files  data  (who  what  when): 

•  Engagement  times  (when) 

•  Engagement  classes  & 
outcomes  (what) 

•  Engagement  parties  (who) 


Dependencies 


•  Latency: 

•  difference  between  event  appearance  and  event  execution 

•  Throughput: 

•  number  of  executions  per  time  (attacks,  found  entities) 

•  Gain/ reward: 

•  aggregated  value  from  execution 

•  Defensive  &  offensive  scores: 

•  number  of  enemy/friendly  attacks/ destructions 

•  enemy/friendly  casualties 

•  Resource  utilization 

•  number  of  engagements 
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Scenario  Assumptions  & 
Simulation  Setup 


Scenarios: 

■  Missions  vetted  with  SMEs,  various 
tasks  explored 

-  Conduct  area  reconnaissance 

-  Enter  building  &  clear  a  room 

-  Establish  an  observation  post 

-  Patrol  a  route 

■  Vignettes  within  missions  (responding 
to  high  risk  threats) 

-  lEDs,  VBIEDs,  snipers,  rockets, 
mortars,  small  arms  fire,  insurgent 
activities,  etc. 


Measures: 

■  Determined  DDD  and  field  experiment 
overlaps,  independent  metrics 

-  DDD  can  not  test  battery  life  of 

equipment,  field  experiments  can  not 
test  all  possibilities 


Technologies/TTPs: 

■  Determined  relevant  attributes  which 
can  be  measured 

-  i.e.,  average  time  to  clear  a  room  and 
average  rate  of  movement  using 
unattended  sensor,  MAV,  etc. 

■  Obtained  TTP  for  employment  of 
technologies 
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Uncertainty,  Risk,  Complexity  High 


zr 

APTIMA 
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Scenarios  and  Simulation 

Examples 

£ 

O 


> Fortified  checkpoints  (CPs) 
>Guard  towers  (GTs) 

~  600  troops  to  man  CPs  &  GTs 
~12K  Iraqis  live  inGZ 
-8  CPs  in  GZ 
~  30-36  troops/s hitt  at  CPs 
~  2500  vehicles/day  thru  CPs 
-  6500  people/daythru  CPs 
~  20  GTs  in  GZ 
~  1 0  US  &  Iraqi  troops/GT 

Col  Bobby  Campbell.  GZ  Command*/ 


Dismounted  Patrol 


> Roving  patrols 
>Ad-hoc  checkpoints 
>PIR/IR  dnven 

~  36  patrots/day  around  GZ 
~8-10soldieis/patrol 
~  2-6  hours  mission 
-  Typically  poor  Intel  and  SA 
Col  Bobby  Campb*b.  GZ  Command*/ 


Cordon  and  Search 


> Recon  &  secure  route 
>  Isolate  &  cordon  off  area 
►Contain  &  block  access 
> Search  &  back-clear  area 
> Maintain  area  security 
—  1 00  Iraqi  buildings 
~  400  troops  to  exec  mission 
~  1 2  hrs  mission  complete  time 
-Assumes  excellent  intel 

Co I  Bobby  Campbell.  GZ  Command*/ 


Defensive  Mission  Type  Offensive 
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Tech 


- \ 

■  Challenge: 

-  Will  utilizing  sensing 
technology  to  secure 
buildings  free  up 
troops  to  accomplish 
other  tasks? 

/ 

■  Scenario  Parameters: 

-  2  sensors  per  buddy  team 

-  Utilized  on  average  2  sensors  per 
building 
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Average  #  of 

Engagements  per  BT 


Example  Question  : 
>qv/workload  effects 


■  Comment: 

-  Technology  allows  alternative 
resource  employment  to  reduce 
load  and  increase  number  of 
executed  tasks 
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Average  #  BT  per  Platoon 


Analysis  Example  : 
Potential  Workload  Reduction 

=rM=  APTIMA 

yj=\  \y  HUMAN-CENTERED 

J4.  yf  ENGINEERING 

Assessment 

■o 

0) 

O) 

(0 

U) 

c 

0) 


Troop  Attacks,  Searches,  and  Security  Operations 


0  12  3  4 

#  of  Engagements  (Search,  Secure,  Attack,  Monitor,  IED) 

per  BT 


Combinations  of 
technologies  help 
reduce  troop 
workload 

■  W/OTech. 

0  W  Tech. 


•  #  of  BT  per  platoon:  12 


I  mplications: 

•  Generate  analysis  of  effect  of  technologies  on  casualty  reduction 

•  Field  tests  for  alternative  manning  employment  with  technologies 
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Benefits  of  Agent-based 

J4.  yt  ENGINEERING 

Approach 

■  M&S  Value 

-  Can  fulfill  a  key  complementary  role  in  the  testing  and  evaluation  of 
technologies  and  associated  TTPs  for  C2 

■  Model  Validation  and  Enhancement 

-  DDD  models,  parameters,  measures,  and  scenarios  are  adjusted  to 
account  for  that  which  is  learned  in  field  experimentation 

■  Testing  Technology  Integration 

-  Has  the  potential  to  test  technology  integration  concepts  before  they  are 
inserted  in  the  field 

■  Sensitivity  Analysis 

-  Has  the  potential  to  explore  through  sensitivity  analyses,  the  effects  of 
performance  improvements  of  the  technologies  on  key  MOP/MOE 

■  Field  Experimentation  Focus 

-  Can  support  experiment  design,  staffing,  technology  use,  & 
measurement  and  focus  field  studies  on  most  relevant  parameters 
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Summary 


■  Purchasing  and  funding  decisions 

-  In  terms  of  a  viable  solution,  agent-based  simulations  predict 
that  the  technologies  can  improve  local  mission  performance 

■  Engineering  decisions 

-  In  terms  of  performance  objectives,  simulations  provide  baseline 
predictions  of  magnitude  of  performance  improvements 
expected  from  these  technologies 

-  In  terms  of  reliability,  simulations  assess  different  tasks  and 
missions,  providing  insight  about  circumstances  that  may 
challenge  systems 
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Questions/Comments 


Questions? 


For  more  information  contact: 

Stacy  Lovell 
Aptima,  Inc. 
781.496.2462 
slovell@aptima.com 
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